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Development Overhead 

Overhead = damage over time to your game 
Time spent 'not making the game' is wasted time 

Reduce that time! 

- Audit the development pipeline 

- Identify high ROI candidates to target 



AAA Games have become huge 

More, bigger assets 
More asset dependencies 
More asset churn 

Longer offline asset processing times 


Greater authoring overhead for individual assets 


Asset Size (lOx increase) 



■ F.E.A.R. 2 (2009) 
SOM (2014) 
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Asset Count (7x increase) 
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■ F.E.A.R. 2 (2009) 
SOM (2014) 
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Asset Type Breakdown 


Size 



w Textures 
Sounds 
■ Other 
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Size ( - texture, sound) 



~ Animations 
Worlds 

■ Models 
Database Records 

■ Behaviors 

' 

■ FX 

■ Shaders 
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Intro 


Asset Type Breakdown 
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Asset Type Breakdown 


Count (- Sound, Database Records) 



* Textures 
Animation 
w Worlds 

■ FX 

■ Models 
Behaviors 

■ Shaders 



1. Staying in Sync 


Ch 1. Staying in Sync 


• Latest Build 

- Game, Tool suite, etc 

- Synchronization, Installation 

- Pre-reqs 

• Latest Assets 

— Synchronization 

- Source, Packed Assets 

• This began to take considerable time 
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1. Staying in Sync 


Definitions - Assets 


• Assets 

- Source 

• Written and read by tools 

• Contains authored content 

- Packed 

• Written by packing process, consumed by runtime 

• Contains content loaded by game 


1. Staying in Sync 


Update Frequency 


• Frequency 

- ~Daily 

• Extenuating circumstances 

- Problem with 'released' build 

- Blocker fixes 

- New / Fixed content 

- New / Fixed code 
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1. Staying in Sync 


Automated Syncing 


• Pros 

— Offline time is free 

• Cons 

- Nightly builds may not be finished 

- Requires infrastructure 

• Stagger to avoid server hit 

• 'Offline' is relative 
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1. Staying in Sync 

Average sync times (6x faster) 



Sync (min) 
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1. Staying in Sync 


Sync Faster 

• Ensure sufficient network speed 

- Network pipe truncated 

- Old, bad switch 

- 360 network cable (2 twisted pair) 


Can be detected on PC 


1. Staying in Sync 


Sync Less 

• Deterministic file format 

• Processed asset files are 1:1 - produced the same way 
every time 

• Don't check in no-op revisions 

• Do not embed 

- Timestamps 

- Unseeded (new) GUIDS 

- Absolute Paths 
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1. Staying in Sync 

Churn reduction (lOx smaller) 



w Revert Unchanged 
Processed 
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1. Staying in Sync 

Automated Build Deployment 

♦ Pros 

— More Time Saved 

• Cons 

— May need to reboot 

• Pre-reqs 

— What if binaries are still open 

• Potentially unsaved work 

— Significant state change, risk 
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1. Staying in Sync 

Takeaway - Staying in Sync 

• Candidate for automation 

• Sync Faster 

— Ensure optimal connection 

• Sync Less 

— Deterministic format 


Ch 2. Load Lag 


2. Load Lag 


Loading Files to Author 

• Began to take a long time 

- More data to load 

- More complex assets, split over multiple files 

- More complex operations 

• Objective here is to get to edit as quickly as 
possible 
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2. Load Lag 


Load Smarter 

• Examine the source file format 

• Emphasis on minimum footprint, maximum 
load speed 

- How fast can you get the data into memory (Disk) 

- How fast can the data be interpreted (CPU) 


2. Load Lag 


LTA Source file format 


• LTA - Lith Tech ASCII (xml like text) 

- Human Read-able 

- Large file size 

- Slow to Interpret 

• Encoded, Compressed ASCII 

- Smaller on disk 

- Faster to get in memory 

- Slower to Interpret 
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2. Load Lag 

LTA Source file format 

• Compressed Binary representation 



- Smaller on disk 

- Far faster, no need to parse text 

- File tree roots compressed independently (Zlib) 

- Load / decompress in parallel, or partially 

- CRC checking exposes file corruption 

- Provide utilities to convert to human read-able format 
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2 . Load Lag 


Footprint on disk (lOx smaller) 
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Load Time (lOx faster) 
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Load on demand 


• Don't load most data until it is needed 

- Requires user actions to prompt additional loading 
— Ideal for self contained workflow 
— Tree controls work well for this 

• Requires appropriate granularity of source files 

— Tough to do with a single file 



2. Load Lag 


Delay Load Times (15x faster) 



Production Level(sec) Character Animation(sec) 


w Time to open 
Fully loaded 
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Load in the background 

• Only load some of the data up front 

— Let the user begin editing when the initial chunk 
has loaded 

- Data needed, but not immediately 

- User blocks if necessary 


Ex. Visual Assist, Visual Studio Intellisense 



2. Load Lag 


Background Load Times (5x) 



Talion Behavior (sec) Ai Behavior (sec) 


■ Time to open 
~ Fully loaded 
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2. Load Lag 

Death of a thousand cuts 

• Need to load a ton of little files 

- Hard drive is bad at this 

• Asynchronous 10 will help 

• Build a 'checkpoint' 

- Single file, compressed for minimal footprint 

- Patch in cases where its out of date 

- 90k files / 800mb to checkpoint at 30 mb 



2. Load Log 


Load Time (20) 
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2. Load Lag 


Cheat (quick wins) 


• Disk - SSD 

— Quick wins for serialization 

- Smaller Capacity 

• CPU - Thread Pools 

- Parallelize Implementation 

- Not applicable everywhere 

- Complicated, comes with overhead 
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2. Load Lag 


Takeaway - Load Lag 

• Load Smarter 

- Reduce Disk, CPU costs 

- Be creative 

• Defer Loading 

- Temporarily, Indefinitely 

• Cheat with hardware 


Ch 3. General Performance 


3. General Performance 

Nothing is future-proof 

• There are always areas that scale poorly 

- Not realistic to support every future path 

• Plan to spend time 

- Identifying points of failure 

- Invest resources addressing these 
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3. General Performance 


Profile your pipeline 

• Windows Performance Toolkit 

- Bruce Dawson has several talks 

• Fast enough to run continuously 

• Covers all applications 

• User events to provide context 


Performance Reports 

• Help users provide feedback 

• Gather high level metrics 

• Identify / analyze pain points 

• Detect unexpected use 


3. General Performance 


Xperf Graph Image 


I* C:\YEAH.etl - Windows Performance Analyzer 
■ File Graphs Trace Window .Help 




CPU Sampling by Process 
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CPU Sampling by Thread 
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3. General Performance 


Xperf Images #2 
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3. General Performance 


Heap tracking 

• Xperf does this as well 

- Much slower, application specific 

- Generates far more data 

- Captures every allocation 

• When Ram is paged, the disk suffers 

- Use less memory 




3. General Performance 


Xperf Heap Image 2 


- [58 .831643768 s - 59.060338204 s] - 0.228694436 s - Windows Performance Analyzer! 


Help 


[Root] 

|- ntdll.dlllRtlUserThreadStart 
kernel32.dll!? 
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3. General Performance 

WMI provides useful data 

• Windows Management Instrumentation 

• Exposes 

- Hardware specifications 
— Hardware utilization details 

r 7 
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• WMI Implementation Details 
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3. General Performance 


WMI Exposes 

• Processor Information 

- Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (6p, 121 
cores) 

• GPU Information 

- NVIDIA Quadro K5000 

- Dedicated 66.99 MB / 3.94 GB (in use) 

- Shared 3.84 GB / 7.42 GB (in use) 
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3. General Performance 


WMI Exposes 2 

• Physical Memory 

-12.63 GB /15.92 GB 

• Page File Info 

-2.90 GB /15.92 GB 


3. General Performance 


WMI Exposes 3 

• Network Adapter Information 

- lntel[R] 82579LM: 1000 mbps 

• Volume Information 

- C:\ - Available (51.60%) 240.33 GB / 465.75 GB 

- D:\ - Available (43.90%) 408.89 GB / 931.51 GB 


3. General Performance 


WMI Exposes 4 

• Process Specifics 

-CPU Time 

- System Memory footprint 

- Dedicated VRAM footprint 

- Shared VRAM footprint 


3. General Performance 

Thread Pools 

• Reduce time waiting on CPU 

• Analogy: moving 

- Faster with friends 

- But only if they actually work 



3. General Performance 


Thread Pools 


• Pros 

- Can yield linear speed improvements 

- Fairly easy to drop in if appropriate 

• Cons 

- Will not offset poor algorithm choices 

- Core contention 

- More complicated 

- Only as fast as the slowest job 
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3. General Performance 


Thread Pools 


• 5 jobs done in sequence on a single core 

Time 

( II I I I f II I 
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3. General Performance 


Thread Pools 

• Same 5 jobs performed on 4 cores 

Time 
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3. General Performance 

Database Find Time (7x faster) 



Search time (sec) 
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3. General Performance 

Takeaway -General Performance 

• Profile your pipeline 

— What is the issue? 

- Don't guess and hope 

• High level information also useful 

Ch 4. Asset Processing 
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4. Asset Processing 


Asset Processing 

• The transformation of an asset so it can be 
consumed by the target 

• We call this 'Packing' 

- Known as baking, cooking, etc 

• Time spent here on SOM went through the roof 


4. Asset Processing 


Where is the time going? 

• More assets to pack 

• More targets to pack for (5 platforms) 

• More offline processing (arms race) 

• Many of the assets are larger, more complex 

• Serialization 

• Synchronize, Commit 



4. Asset Processing 


Main use-cases 


• Bulk Asset Reprocessing 

- Packer has changed 

- Format has changed 

- Source assets have changed 

- It has been scheduled 

• Iteration on authored content 

- Intent is to see changes in the runtime 
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4. Asset Processing 

Content Iteration 

• This needs to go quickly at all costs 

- No different than compile / link times 

• Allow a user to only pack the things affected by 
their change 

• Often means supporting more discrete outputs 


4. Asset Processing 

World Pack Time (7x faster) 



■ All Outputs 
World Geometry 
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4. Asset Processing 


Flexibility > Speed? 

• Runtime structures typically optimized for speed 
not flexibility 

• Consider allowing non-optimal formats to 
facilitate iteration 

• Ex. GDB Patch Database 

- Consumed more memory 

- Improved iteration lOOOx 



4. Asset Processing 

Packing Database (lOOOx faster) 
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4. Asset Processing 


Bulk case - Process pools 

• Pros 

- Win of thread pools without complexity or headache 
of multi-threaded code 

• New environment 

- More resilient 

• Crashing process self-contained 

- New address range 



4. Asset Processing 


Bulk case - Process pools 

• Cons 

- More Overhead 

• Slow to spin up 

- No shared memory 

• More serialization 
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- Requires more infrastructure 

- Requires inter-process communication 
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4. Asset Processing 


Asset Builder 


^ \ Master Exe 


Worker Exe | j 


1 World P. 1 
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1 World Packer I 
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4. Asset Processing 

More power, more problems 

• As cores increase, new problems 

- Load on RAM increases 

- RAM per core matters 

- Load on Disk increases 

• Splitting Across Machines 

- New disk to saturate / synchronize 

- New environment to synchronize 

- Network bandwidth an issue 
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4. Asset Processing 

Takeaway- Asset Processing 

• Content Iteration 

- Take as little time as possible 

- Process only what has changed 

• Bulk Reprocessing 

- Farming more hardware requires infrastructure 

Ch 5. Content Dependencies 
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5. Content Dependencies 

Content Dependencies 

• More Content than ever 

• Establishing intentional relationships is 
important 

• Want to be as efficient as possible with 
content changes 

• Re-use = less work 



5. Content Dependencies 


Asset Encapsulation 


• 'Has a' relationship 

• Formally wrap common data as reference-able 
prefab 

• Change def. once, it propagates 

• Can't forget to change it in multiple places 

• Complications when 'slightly' different 
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5. Content Dependencies 

Diagram - Encapsulation 
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5. Content Dependencies 


Data Inheritance 


• 'Is A' Relationship for data 

• Parent is an instance rather than a schema 

• Allows the adoption of a subset of values 

• Property sheets work well here 


5. Content Dependencies 


Diagram - D 
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Inheritance 










5. Content Dependencies 


Combining The Two 

• Ex. Rock Prefab in a world 

• Stamp it down in the world a few times 

• Instances have a unique transform 

• What if prefab has another property we want to 
adjust per instance 

- Specific properties 

- Supports unique game-play 


5. Content Dependencies 


Combining The Two 

• The Instance needs a means of influencing the 
implementation 

• 1. Asset internally provides multiple 
implementations that can be selected by the 
instance 

• 2. Asset exposes an interface which is 
implemented by the instance 
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5. Content Dependencies 

Ex. Nested Composition 

Perspective 


\ 
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5. Content Dependencies 


Ex. Instance Overrides 
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5. Content Dependencies 


Enough Rope 

• A deep prefab tree is intimidating, but is ideal for 
maximum re-use 

• Tons can be updated with a single change 

• Viewing asset relationships is essential 

• Invest in visibility 

- Make it less complicated 



5. Content Dependencies 


Asset Dependencies 


Depends On Used By Properties 

J ' O Pref abs\Kits\S hantyKit\S hantyS tackGb. PrefabOGc 
J ■■ Material 

pjj materials\0dev\flatColors\flatPink.Mat06c 
J " PS materials\default.mat06c 
J ■■ Record 

■■ , J surfaces\surfaces\default 
k --(Jp ShaderTree 
J -Jy texture 

■■ , J materials\Odev\debug\test\T estN ormalM ap. T extureOGc 
J materials\Odev\flatColors\flatWhite.TextureOGc 
J materials\0dev\grids\grid25G.Texture06c 

Prefab 
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5. Content Dependencies 

Hedge your bets, plan for issues 

• Mistakes will be made 

• Asset Validation 

— Pack time validation 

- Client asset validation pre-submission 

- Server asset validation post-submission 

• Try to have things that detect problems for you 



5. Content Dependencies 


Recap 

• Overhead = damage over time to your game 

• Invest in productivity 

- Profile development pipeline 

- You may be surprised 

- Seek out low hanging fruit 

- Establish / expose asset relationships 




MONOLITH IS HIRING! 


www.lith.com/Jobs 


Q& A 


• Doug Heimer - Lead Software Engineer 

• doug.heimer@lith.com 

• MONOLITH IS HIRING 

— www.lith.com/Jobs 


SMADOW^MORDOF 


